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To aid institutions in identifying and using 
inforaation about educational “outcomes, the National Center for 
Higher Education Management Systems (NCHEMS) develored an "outcome 
structure," a new system for organizing outcomes and outccaes 
inforrgation for purposes of classification, analysis and 
decisionmaking. Preliminary tryout and review of this: structure 
suggest that it has potential use in: stisulating people to realize — 
the importance of having informaticn about educaticnal outcomes; 
stimulating people to think more systematically and concretely about 
what they are trying to accosplish in their institutions and progranss’ 
(and fof whom); helping institutional officials. to identify 
educational needs, develop goals, .translate goals into acre concrete 
objectives, plan for outcomes, elevate the institution and its “A 
program; and imppoving communications with clientele and ccncerned 
publics. One of the projects used to try out-the NCHEMS Outcomes 
‘Structure is reported. It was conducted in the winter.of 1977 by the 
University of Colorado at Boulder. (Author/SPG) 
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USING THE NCHEMS OUTCOMES STRUCTURE. TO TEST THE ADEQUACY 
OF OUTCOMES LISTS: TRYOUT AT A STATE UNIVERSITY 


~ 


, Oscar T. Lenning 
Jean, J. Endo _ National Center for Higher 
Uulverstty of Colorado,,Boulder | Education Management Systems . 
e . ‘ Z P| 
. All of the major activities conducted by administrators, faculty, and 


other professional staff members at postsecondary institutions are, presumably, © 


aimed in some way toward bringing about “educational outcomes ." “In this "age. 


t 


of sass eae college AONLOSSTeanOUs and others. have become especially™ 


concerned about concretely identi fying and understanding the impacts of their - 


institution on students, the community, and society. After two years of concen-. ~ 
trated effort, the National Center for Higher Education Management. Systems - 
(NEHEMS) developed an “outéories structure," a new system for organizing outcomes 
and outcomes information for sinpdees of classification, analysis and decision- 


making (Lenning, Lee, Micek, and Service, 1977). Preliminary tryout, and review . 


_ of this structure suggests that it has potential use in: (1) stindlating ‘pegple | 


’ 


to realize the importance of having information about educational outcomes; ‘ 
(2) stimulating, people to think ‘more systematically aad concretely about what 

they are trying to accomplish in their institutions and programs (and for whom); 

(3) helping. institutional officials to identify educational needs, develop goals, ~ 
translate goals into more concrete abdeetivess plan. for the outcomes , evaluate 


the institution and its programs; and (4) “forovine communication about outcomes 


with clientele and seieeeiat publics; and so forth (for example, several students 


and student personnel administrators interviewed at a couple of small colleges 


- felt that the Structure could also be used to assist students in be i what 


they want to accomplish for themselves during college). . # 


pts is a report of one of the projects used to try ut the NCHEMS Outcomes 


aati Ae 
Structure in a preliminary way. It was ‘conducted. in the. e finter of 1977 by the © 


*.. 


‘University of Colorado at Boulder. Over a period of several years, through. 
surveys and interviews of important clientele groups, several extensive lists 
of intended outcomes were developed. by staff of the Office of the Vice Chan- 
cellor for Academic Affairs on the Boulder Campus’ that were conprehens ve for 
thet planning needs. The “NCHENG Outcomes Structure was ‘sppTied to these 

; lists in a way that would reveal the adequacy ‘of the coverage of those lists-- 
using step-by-step procedures that had been developed at NCHEMS (Lenning, 1977). 
This process revealed several outcomes areas considered to be important that 
had been overlooked in developing the lists. - The lists were modified, along 
with the Freshman Questionnaire which} ihad been based in part. on the lists. 

The eeuathder of this paper will go into detail about this project and. its 


results. 


THE NCHEMS OUTCOMES STRUCTURE 


The purposes of the Outcomes Structure have been outlined in the introduc- 
tion to the paper. The Structure consists of three dimensions along which 
outcomes or information abaut outcomes can be placed and related to one another. 
The structure ‘is based on a conceptual franewar’ that defines’ six attributes 
of educational outcomes in postsecondary educatio plus five other factors that 
are important for understanding particular outcomes (Lenning, Micek, and Service, 
1978; BENG Lee, ‘Micek, and Service, 1979). The three dimensions of the 
structure are described below: 

0 * Audience -- The "audience" dimension focuses on who or what receives or is 
affected by the outcome of concern, or is intended to receive or be 
affected by it. It has five broad categories, and subcategories for 


them; as outlined in Figure 1. 
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‘O _ Type of outcome -- The Ai ypceatesutooed” dimens ton focuses on whether the 


, i. ag Theg 


. 
¢ 


* 
outcome results in maintenance (staptttzetton, reproduction, /or: 


preservation) or change (reorganization, modification, revision, or 


replacement), and on the basic entity within the audience that is 


maintained or changed. This dimenston ‘also has five broad cate~ 

gor es, and each is subdivided into categories and subcategories of 
se more detatl and specificity, *as outlined in Figure a5 
Standard definitions are provided for every. category and subeategory 


of this dimension. stone with: ‘illustrative examples of outcomes 


“measures and ‘indicators for each. 


The “time” dimension focuses on when the. outcome. is expected to 
or does occur, and on how long the outcome persists. ~ The categories 
and subcategories for.the "time" dimension tha “are deemed most a 
Pere nerd depending on the audtahen of concern, on the philo: 
sophy of the person using the Structure, and on the context in which 
the Structure is being used. (For example, its use at the institu- 
tonewide level may very well reniire different time categories than 
its use at the institutional program level .) To illustrade this, 
two quite different student. outcome sets nf categories that could be 
used for the "time" dimension are shown in Figures 3 and 4. The one 
presented in Figure 4, is the time classification used by the Uhiver- 
sity of Colorado at Boulder to follow the chronological path taken 
by new freshmen through their educational careers. Identification 
of the times when different data should be collected is valuable in 


Tanning. 
planning 6 Bah 
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Figure E 
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‘ AUDIENCE DIMENSION OF THE NCHEMS OUTCOMES STRUCTURE? 
10. . Individual/Group Clients—This category reto’s to parsons of grouns of persons who are diroct Qients of the postsecondary education unit of 
Concern and/or their inmouiate associates, such as family and relatives of peors, 
". Students —Individuals or groups of Individuals who currently are enrotlod In the program, Institution, or system of postsecondary education. 
© 92 Former Students —individuats or groups of Individuals who formerty wore enrolled in the program, Institution, Qe system of postsecondary 
@dugation. « Pas ee c : 
os 13. Family and Relatives of Students of Former Students 
. 14, Peers end Associates of Students or Former Students ‘ 
‘ . : 
18. Feculty . 
a 4 . " , 
, 16. Statt Other then Feculty ; 
/ 17. Other individual/Group Clients—An example would be an Individual who Is none of the above but Is sorved by an advisory service olfered 
ese, ~ by the college. * rae et 
/ 20. interest-Based Communities—This category reters to large groups that are Identified as entities working toward a well-defined Interest or ’ 
yi mussion, t , . 
U / 3 » . 
: :21, Private Enterprise Communities —Communities where a major purpose Is financial remuneration and profit—for example, corporations, / 3 
small businesses, and farmers. . : + 2 
& vw’ 22. Association Communities—Communities where members bolong on the basis of affiliation rather than employment, such as unions and. 
professional societies. 
. é . h 
23. Government Communities ~Communities designed to administer government regutations and services, such as city hall, stale department 
of education, ang legistative communities. : 
7 24. Nongovernmental/Public Service Communities Other than the Institution Producing the Outcome—Nonprotit service organizations, such 
83 schools, hospitals, waifare agencics philanthropic foundations, colleges (other than the college producing the outcome), and resoarch a 
organizations. a ' . 
25. Institution of Institutional Unit Producing the Qutcome—The postsecondary education Institution and/or units within that instltution that 
are perceived as the producer/facilitator of the outcoma(s) of concern. : 7 
i } = : 
, 26. Other Interest-Based Communities —An example would be an ad hoc coalition task force of representatives trom two or more of the above 
areas. é ‘ 
. ° ‘3 : ’ 
3%. Geographic-Bases Communities —This category relers to large groups declined on the basis of functional territorial boundaries. 4 
; 31. Local Community—A township, city, county, metropolitan area or otfier type of locality having particular boundaries. It is not necessarily 
; festricted to the legal of jurisdictional boundary, but tne functional one in which the impact of tha institution is (of should be) directly and 
; orn physically felt. The boundaries will vary with the institution/ program and outcome of concern. a 
32. The State 
33. A Region—An aggregation of states or parts of states. 
4. The Natian 
ry 35. Aninternational Community ; ' 
° ae y . 
36. Other Geographic-Based Communities—An example would be & research discovery that affects primarily people living In the coldest 
latitudes, or where it snows heavily. ‘ . : 3 
: 40. Aguyregates of People—This category Yeters to sudpooulationgs of peosie distinguished by particulge characteristics {hat may indicate common 
concerns, aceds or wants, but who do not necessarily have a common interest of Missidn, and therefore do Not constitute Communitics, 
’ . riba 
41. Ability Level Sudpopulations —Subpopulations defined accoruing to level ‘of ability/proficiency on generat intellectual tunctioning of oe, 
Spocilic skuls—for wxamplo, gifted, typical, disadvantaged, or skilled, semi-skilled, unskilled, é 
7 
42. Age Sudpopulations RE, aa : : . 
43. Educational Level Sudpopulations : 
44. lacome Level Subpopulations a 
45. Occupation Subdpopulations ; ; - t i 
; 468. Physical Disadility Condition Subpopulations . ; _ 8 
47, Race Subpopulations 
+ 48. Sex Sudpdpuletions rid 
o 49, Other Such Aggregates ; 
: j ® - , 
$0. Otner Audiences —Fxampins would be tho Natural onvironment tt ut Is affoctod by univorsity-sponsored rosearch (which In turn would be 
@xpovted to Nave lnpacts on audioncos such as Individuals and oe munitios) and populations of anynuls (such as tho animals affoctod by 


olforts ty hovp doplutud Spuyjos fron becoming oatinct of by the Uusulug ment of vutulinary mudicinos). 


Reprinted from Lenning, Lee, Micek, and Service (1977), page ah. ° 
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Figure 2 \ a - 
_ FOCUS CATEGORIES AND SUBCATEGORIES "IN THE Ng \s : 
TYPE-OF-OUTCOME DIMENSION OF THE NCHEMS OUTCOMES STRUCTURE® ~ ‘| 


Category 


Entity Being Maintained or Changed Code Number 


ree 
Entity Being Maintained or Changed 


1000 ECONOMIC OULSOMES 


1100 Economic Access and Independence Outcomes 
1110. Economic Access 
sn Economic Flexibility, Adaptability, and Security 
Income and Standar& of Living 


2000 - HUMAN CHARACTERISTIC OUTCOMES eontitued) 


2760 Power and/or Authority 
2770 Job, School, or Lite Success 
2780 = Othor Status, Recognition, and Caitification Outcomes 


2800 Social Activities and Roles 
Adjustment to Retirement 
Altiliations 
Avocational and Social Activifles and Rolas 
Career and Vocational Activities and Roles 3 
Citizonship Activitics and Roles 5 


1319 Economic Productivity and Production Family Activities and Roles 


; 4 Friendships and Relationships 
’ 1320 Economic Services Provided Other Activity and Role Outcomes 


“1400 Othier Economic Outcomes 2900 ' Other Human Characteristic Outcomes 


\ 


¥200 *eoniomie Rosources and Costs 
1210 Economic Costs and Efficiency 
1220 Economic Resources (including’employees) 


1300 Economic Production 


2000 NUMAN CHARACTERISTIC OU tM ES 


2100 Aspirations 
2110 Desires, Aims, and Goals, 
2120 Dislikes, Likes, and Interests 
2130 Motivation or Drive Level 
2140 Other Aspirational Outcomes 


"3100 General Knowledgerand Understanding - 
3110 Knowledge and Understanding of General Fag 
Terminology 
3120 Knowledge and Understanding of General Proces 
3130 Knowledge and Understanding of Genaral Theory 
Competence.and Skills 3140 Other General Knowledge ana Unddratanaing 
2210. Academic Shills Specialized Knowledge and Undorstanding aoe 
2220 Citizenship and Family Membership Skills 3210 Knowledge and Understanding of Speciatized 
2230 Creativity Skills : afd Terminology 
2240 Expressitn and Communication Skills” 3220 KnoWledge and Understanding 6f Spocialized 6 aaa 
_ 2250 Intellectual Sktils ec Processes 
2260 = Interpersonal, Leadership, and Organizational Skills 3230 Knowledge and Understanding of Specialized Theory 
2270 Occupational and Employability Skulls 3240 Other Specialized Knowledge and Understanding 
2280 Physical and Motor Skills - 
Réscarch and Scholarship 


; hor Skill 
Revd. Otay Sill Oateuaes ; 9310 Research and SchPlarship Knowledge a 
Morale, Satisfaction, and Affective Characteristics ~- Understanding 
ae ene e and Values _ iii 3320 Research and Scholarship Products 
eliefs, Commitments, and nilosophy 0 Life : 
2230 Féclings and Emotions an ag Rata ap lh \ 
2340 Mores, Customs, and Standards of Conduct 


Dance 
2350 Other Aticctive Outcomes Debate and Oratory 


Perceptual Characteristics : ) Drama 

2410 Perceptual Awareness and Sensitivity Literature and Writing 

2420) Perception of Self ' Music 

2430 Perception of Others Painting, Orawing, and Photography 
2440 Perception of Things Sculpture. 

2450 Other Perceptual Outcomes Other Fine Arts n i 


Personality and Personal Coping Characteristics 
Adventurousness and Initiatwe 
Autononvy and Independence 
Dependadility ano Responsibility 
Oogmafic/Open-Minded, Auihssitaran/Bomeersiic 
Flexibilityyand Adaptability * 


4000 RESOURCE ANDO SERVICE PROVISION QUITCOMES 


4100 Provision of Facilitics and Eveats a ore 


‘Habits 

Psychological Functioning 

Tolerance and Persistence 

Other Personhlity and Personal Coping Outcome$ 


Physical and i Le aay Characteristics 


2610 Physical Fitness and Traits 
2620 Physiological Health 


4110 Provision of Facilities 
4120 Provision or Sponsorship of Events + 


4200 Provision of Direct Services 


/4210 Teaching % 
4220 “ Advisory and Analytic Assistance 
ss 4230 Treatment, Care, and Reterral Sorvices 
4240 Provision of Other Services 


2530 Other Physical or Physiological Outcomes 4300 Other Resource and Service Provision Outcomes 
he 44 


Status, Recognition, and Certitication 4 
2710 Completion or Achievement Award 5000 OTHER MAINTENANCE AND.CHANGE OUTCOMES 


Credit Recognition. 5100 “Acsthetic-Cultural Activities, Traditions, and Conditions 
Imago, Reputation, or Status . ; 
5209 Organizational Format, Activity, and Operation 


Licensing and Certification 
Obtaining a Jot of Admission to a Follow-up Program §300 Other Maintenance and Change hes 


aReprinted from Lenning, Lee, ‘Micek, and Service (1977), page 27. The fourth-level 
Categories, into which any of the categories listed here can be divided, are 
"maintenance" (a fourth digit of "1") and "change" (a fourth digit of "2"). 
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Figure 3 | 2 
ONE POSSIBLE SET OF STUDENT OUTCOME’ = 
CATEGORIES FOR THE TIME-DIMENSION® = 


% = 


11. Short-duration 12. Short-duration 


outcomes appear- outcomes appear- 

ing at or prior to ng after gradu- 

graduation” ation 
i \ 
a ‘: 

\ F 
21.. Long-duration 22. Long-duration . \ 
outcomes appear- outcomes appear- | ~ a? 


ing at or prior ing after gradu- 
———— to graduation : ation 


as: 
Ne. 


F : * * hs ‘ ‘ rs SS 
. "Reprinted from Lenning, Lee, Micek, and Service (1977), page 29% 
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Data collecte 


Figure 4 
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_ TAXONOMY OF TIMES TO CONSIDER FOR COLLECTING UNDERGRADUATE, =, 
STUDENT OUTCOMES DATA USING SURVEY QUESTIONNAIRES? 


s 


Prior to Fall-Registration 
One Month Following the First Day of Classes 


Data cqllecte 


in the Fall 
Data collected One Month Prior to First Semester Final Exams 
Data collected One Month After Spring Semester Classes Begin 
Data collected One Month Prior to Spring Semester Final, Exams 
Other, For Example, In the Middle of the Summer Term 


‘ 


120 ' Data From Sophomore Students... ‘ “ 


121 bata Collected Prior to Fall Registration 
et 122 Data. Collected Oné Month Following the First Day of Classes, 
4 an in the Fall. 

. 123 Data Collected One Month: Prior to First Semester Final Exams 
, 124 Data Collected One Month After Spring Semester Classes Begin 
- 125\ Data Collected One Month Prior to Spring Semester Final Exams 

126 Other, For Example, In the Midd]e of the Summer Term 


200 Data From Upper Division Students J . 


- 


210 Data div duntor Students 


- 21] Data collected Prior-to Fall Registration Se. 
Fatal eee sOne Month Following the First Day of Classes 
F oe ° n the Fa 
, 213 Data collected One Month Prior to First Semester Final Exams 
214 Data collected One Month After Spring Semester Classes Begin 
215 Data collected One Month Prior to Spring Semester Final Exams 
216 Other, For Example, In the Middle of the Summer Term 


¢ _ 220 Data From Senior Students 


221 Data Collected Prior to Fall dectsbealan 

222 Data Collected One Month Following the First Day! of Classes 
in the Fall 

- 223 Data Collected One Month prior to First Semester Final Exams 

224 Data Collected One Month After Spring Semester Classes Begin 

225 Data Collected One Month'Prior to Spring Semester Final Exams: 

226 Other, For Example, In the Middle of the Summer Term 


- 300 Data from Alumni 
310 Data Colletted at Graduation . 
320 Data Collected One Year After Graduation 
330 Data Collected Five Years After Graduation 
340 Other, For Example, Data Collected 20 Years After maaan ag 


400 Other, For Example, Data Collected After Students Have “bropped Out" 


®Reprinted from Léenning (1977), page 8. 


tutions. 
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¥ These categories and subcategories 
are based specially on the data collection experiences of staff in the Office of 
the Vice Chancellor for Academic Affairs at the University of Colorado, Boulder. 

Therefore, they may not be entirely appropriate for other postsecondary: insti- 


When the thhee dimensions are put together, they can be pictureg graph-— 


ically as a three dimensional series of cubes formed by the categories for 


2 . 


one of the dimension intersecting each of the categories of the other two 


dimensions. This is illustrated bélow for the broadest categories of each —— 


dimension. 
<a 
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Group Clients 2 ai 
’ ; % 
Interest- . 
Based Clients 
= 
Oo 
~ 
nn 
=z 
S  Geographic- 
© Based Clients 
w 
8 2 
{tt} 
~ 
S Aggregates  . 
of People 
Other - 
Audiences 
E of ; : 
Le ¢ 8 
& xs] E a2 
oo ow _ 6 c 5 
Hf) ee vO vo 
“ <x cw + 
or a oz cw e 
Su aa |e ra . 
ps rt #o- uO sa ow = i) 
“ EE TUE o> & VU = 
‘" 60 coo —- nS as Lo 
cu ot UY Sou o> a 
sf Ess 833 $5 352 
ue Z&bS ZES6 2a on 


TYPE-OF -OUTCONE DINENSION 


‘ 


., It is illustrated at’a more detailed outconé category level as follows, 


‘using’ the category code numbers shown in Figures: 1-3: 


¢ poptence 
 Ve-Family and Relatives of’ 
- - Students and Former 
Students 


nd e 
8 : r t 
; An outcone for the 
ae family and: retatiees 
an , af students and former 
We Oe: _| students that involves 
12.1132.22 -\_°¢ a change in. incom? and ' 
; | - standard of living that 
will) appear. after , 
‘ graduation end be of - 
long duration. 


_ ‘TYPE-OF-OUTCO:E, 
, [1132=Change in Income and, 
; Standard of Living | 


TIME | 

. 22--Appearing after graduaz | 
‘tion and having long’ 

- duration 


When the dimensions are combined at the leyel of detail illustrated above, 
they provide several thousand distinct "cells", The categories within the 
Structure are believed to cover the full range of possible audiences and types | 
of OUECOMES and ‘procedures a ovided for subdividing to even more detailed 
subcategories than provided by the Structure. Thus, when all of the outcomes 
in a list are categorized using the Seriicturs-ied step-by-step procedures for 
this purpose, areas of the Structure that have no outcomes assigned to them 
(or fewer than expected or waited) ‘baconie readily apparent. Then, if desired, 
specific priority outcomes for those areas that seem lacking can be generated 

. _using ‘another set of procedures that have been developed. This, in summary, 
is the process that was used in this tryout of the Structure. 


THE uuIVveRStTY OF COLORADO OUTCOMES Esk AND ITS DEVELOPMENT 


The. Student: Outcomes planning Model is a descriptive model that identifies.. 
the relevant features of a student ' s educational environment: and experience. 
These features.are ‘considered valuable educational expectations by Students and 
the academic commun Fy The Model is “used to guide the collection of infor- 
mation on students that will help in thre planning and management functions of 


an Anstitution, . . Ea See -) ig 


41 


a ss 


This Model’ helped quidé the development of*the outcomes list used by 


the University and is based on the notion that information about. students 
is useful. in planning and nid nieqentan€; In. addition to institutional inputs 


(such as library inne facilities). and student inputs (such as ability), 


. the educational process and. its environment. are the impértant variables which | 


can be altered to produce beneficial educational outcores (such as employment . 
skills, grade point average). These snsts sutton} and student inputs could 
be combined in different educational processes to produce. different levels. 
‘and kinds of outputs. Educational processes include modes and styles of 


‘instruction and course curriculum. The Model evolved at. the’ institution in, 


- 


1975' to assist interna] management, to provide information for external accoun- . ° 


| tability, and academic planning.. Academic planning includes needs assessment, 
program development, resource allocation (budgeting), and evaluation. ae 

- The Model allows for the. systematic collection of student data through | 
the use of surveys. The Motel facilitates, the coordination of basic student- 
related questionnaires and the results can be incorporated into a larger, 
_ more extensive academic planning model (Heyerson and Banfield, 1955)« Other 
‘important items of outcomes information which are available through ‘institu- 
tional records (for example, college entrance exam soores and undergraduate 
‘grade point average) are not included. The Mofle1 is designed to identify 
those items that must be collected directly from students. ; 

The Student Outcomes Planning Model. incTudes students' 'e background, atti- 

tudes, values, aspirations, abilities, and the extent of change in these 
variables. The Model, shown in Figure §, consists of four basic ‘question- 
naires: 1) Freshman Questionnaire, 2) Exiting Students' Survey, 3) Graduating . 
Students’ Questionnaire, and 4) Alumhi Survey. . The todel allows ie’ longitu- — 


- dinal (cohort) and cross- +-sect ional analysis. 
| Developed by-Richard L. Harpel and Jean J. Endo 
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EXITING, STUDENTS 
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+ BRESHMEN 


hay 
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DEMOGRAPHIC CHARACTERISTICS 


Academic ability 
Financial. support 


PREVIOUS EDUCATIONAL “EXPERIENCES 
ASPIRATIONS/EXPECTATIONS/ 
_ MOTIVATIONS 


Reasons for choostng CU 
Highest degree planned 
Career pfans 


/ : 
r 
’ 


oe. 


SELF EVALUATION 


Academic 
Social 


CRITIRAL THINKING ORIENTATION 


"| GOALS/VALUES| 


% 


~LI- 


hy, 


TIME OF DEPARTURE 
REASONS FOR “LEAVING 


FINANCIAL SUPPORT : 


4 


EVALUATAON OF UNIVERSITY | . 
ENVIRONMENT 


—\ Academic © 
Social 


s 


-UNMET NEEDS 


4 


- 
ay . 


FUTURE PLANS 


—- 
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. Figure 5 | 
STUDENT. OUTCOMES PLANNING MODEL 
“FOR . tere © . 
THE UNIVERSITY OF COLORADO--BOULDER = 


. 


GRADUATING STUDENTS ALUMNI 


ACADEMIC ACHIEVEMENT | 3 - . | OCCUPATIONAL SUCCESS 


; ’ Grad 1 ; Positions held 
Bares coe eter — Ski lts. required 
Basic academic skIIIs Assessment of skills, 
Critical thinking : ., dob characteristics 
* orientatton { “¢ Flexibility tn fleld 
i changes 
yer nah ie college , Satisfaction with~ 
oe ob 


; as Satisfaction with 
CAREER DEVELOPMENT |, 


ath 
- Major or field of study 
Highest degree planned 
Importance of Job factors 
Satisfaction with college 
experience: {. 


FURTHER ACADEMIC 
EXPERIENCES” 


Graduate School. 
Continuing education 


PERSONAL/SOCIAL: DEVELOPMENT ‘ 
“Interpersonal skills : "| COMMUNITY ACTIVITY - 


Persona) talents/creativity ; 
_ Apprectation for culture ‘ Clubs /orgatj zations 


Understanding different 


training 


“college preparation 


oe. pis ‘ Mt litary occupational 


SEEN TSAO POLITICAL PARTICIPATION 
: Elections: . 
‘ENVIRONMENT a local activities 
- "  Academtc 
Soctal . CULTURAL INTEREST 
d ‘ . Level of participation 
as ; Breadth of Interest 
a o i ae a 4 | 
‘ ‘ ‘ P j 


- 


“14 


NL’ 


a 
: if 


a 


.the lengitudinal and cross-sectional ‘surveys to determine respon 


“length of the surveys and use only those: instruments needed to 


< af ; 


4 


Longitudinal studies involve testing the same students (cohorts) several . 
times in. their college career with the same instrument. Questionnaires are 
given to students as freshmen, exiting students, iia students, and alum- 
ni. The testing of cohorts sonteole for many external variables such as family 


background and basic ability. Unfortunately, me: needed and useful data\is 


ome 
‘6 ’ 


not produced early in the process. fs 
Cross-sectional studies includes responses : ‘from a sample of students repre- 
“sentative of those at different class levels and are measured at, the same point 
“in time. Change is inferred by comparison babigat classes assuming that the 


students have similar demographic .characteristics. Of course, the removal of 


a 4 ; N F 
dropouts would alter the student characteristics. CrosS-sectional studies are 


... performed on exiting students, graduating Students; -and alumni. eo 


. J $ 
The Model is currently in its third year of t testin --incorporating both 


differences. 
It is hoped that within two years the University will be able to inimize the 
ovide student ; 
data necessary for efficient planning and management a Campus programs. 

The Student Outcames Planning Model centers around 22 student related 
goals and their, measures which are listed in the Freshman Questionnaire. The: 


relative importance of each goal reflects students' expectations while attend- 


ing, the University. Student interests are an Mnparseant component of eucational 2 


| planning. Administrators can balance “ needs of various student populations , 


and the different preferences of vartous educational experts. Several measur- 


able objectives from the Higher Education Measures and Evaluation KIT (c. Robert 


Pace, 1975) were assigned to each goal. For example, the goal e gain self- ° 
“confidence: is: measured by ngpponses to the fol lowing statements: a) "I am ae 
person 6 of worth a on an equal plane with others", b) Bly confidence in myself 
is siisag enough $0 that it doesn't bother me-if. people don' t like me", c) "I 


seem to. have tiner strength in handing things". 
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se determining whether ‘an outcome is in the, desired direction. The exiting | 


$ 
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-Anclude questions that attempt to, measure academic achievement, career develop- 


; and cultural interests. - 


: pocenu USED BY THE GNIVERSITY IN THE CLASSIFICATION OF THE ITEMS ON ITS” 
OUTCOMES LIST: ae 


. Survey will: compare students who have tenborart ly or je a left. the 


“future plans will be. ? : 


Student Outcomes Planning Model, and in part from other sources. ~Although 


‘the primary focus of the list is on student outcomes, a few outcomes for 


\ 
\ 


In addition ta the Pace and Associates KIT, a number of other -sources 


“in the literature were referred: to in devloping the Model” and its assoc- 


lated. “questionnaires. To date, .the Boulder Campus has completed a 1975 
Freshman Questionnaire and a 1977 Freshman Questionnaire and is currently 
developing an Exiting Students’ Sine) to be completed in 1978. The fresh- 
man surveys will be’ administered every tivo years to provide descriptive infor- 


mation and establish trends. Trends in measures of objectives are useful fn 


institution with those who have remained. It will identify the stonouter 
transferouts, and dropouts. ‘It will determine why students Teave the campus, 


the time they left, amount of financial suppor received, and what their — 
The Graduating Students’ Survey will be administered in 1979. It will 


ment, personal/social evelopment, and an evaluation of the university’ Ss environ-- - 
ment. One year later, the alumni survey will measure occupational success, ’ 
further academic experiences, community activities, political participation; 


’ 
- 
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-The University of Colorado outcomes list was derived in part from the -" 


other audiences-are also included, that are not student outcomes but might 


enaditional sources used in developing the Model were Schalman,. et al. 
(1974), Baird (1967), teers (1971), Bloom (1956), Clapp (1946), Cohen and 
and March (1974), Fox (1974), Peterson (1973), Richman and Farmer (1974), = | - 
Trivette (1973), ei and Banfield (1955), and Micek-and Arney (1974). it 
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be expected to relate to student ‘iitcones: for nimple, number Tre 
by faculty, number: of books in the library, and number of eyents and type spon- 
sored by the. departnents or universities. : | 
- Two of the basic pypcesses developed ‘y the NCHEMS statf for using the . ) 

Structure were involved in thisystldy.., - Systematic, straight forward,: step-by- 
step procedures for each of the processes aré provided ‘in Lenning (1977). First, 
“process No." “Qe-classifying outcomes items--was used, to classify all items in 
the: University of Colorado outcomes Vist.’ Then Process No. | eS apple in 

order to evaluate the adequacy of the list, ‘in. terms. of whether important, out- 


_“comes desired by ‘the University community have peel left out. 


a 


PROCESS NO. 2--CLASSIFYING THE OUTCOMES ON THE LIST . M 


The desired "audience" for classification was determined to be c ‘rent and 


graduating students (Code’No. 11). First, each outcome on the University of 


Colorado list was classified in terms of the broadest type-of -outcome categor- 
ies’ Then each was grouped ike the more detailed eateaniries at the second and 
one lévels of detail for that divienston of the Structure, Procedures are 
- provided for addtna additional levels of dedi, using available taxonontes that: 
90, into stil mor detail, for example, Bloom's (1956) taxonomy of the cositive 
densi, or using a ‘Togical array developed locally. At this University it # i 
decided Ahat the third-level Structure “categories were adequate for their ca. 
+ demic planning purposes. | : 
Table 1 shows. the iietverstn0 of Colorado' senor outcome items (right 
half of each page) and the Structure categories that each was classified into 
(veft half of each page). It was doticed. that a number ‘of Structure categories j 
” dia! not have outcomes list items assiqned to them. It was felt that there - 
might be additional outcomes cateoories to which some outcomes from the list 


had: been assigned so Process No. 1 Procedures fol lowed. 
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‘2230 


2250 


2270 


Ct | ie skills 

Attitudes, Values : - Effect of college on 

ee ty e » attitudes and values | 

Perception of Self - ' Self-confidence measure * 

<= Expectations 
Autonomy ‘and Independence. - oe tse Measures of independence 
Tolerance and Persistence ~° Measures-of tolérance and 
Cae . 5 Le persistence : 

Completion or Achievement Award - « Graduation diploma — 

— * § = =~ Special awards gov 5 
’ Licensing and Certification - Percent passed’ specifit 

yg licensing exams . 


Table 1 


THE OUTCOMES LIST ITEMS PLUS a & 
STRUCTURE CATEGORY INTO WHICH EACH WAS PLACED 


"AUDIENCE"--SENIOR STUDENTS AT CU eo a 


NCHEMS Items Included 
Type-of-Outcome 7 on the 
¢. Category University of Colorado. 
Name _ ; Qutcomes List - 
ECONOMIC OUTCOME. (Code 1000) CATEGORIES ef ‘ mn ; 
Income and Standard of Living er Family income we 


[HUNAN CHARACTERISTIC (Code 2000) CATEGORIES) 
ies 3 Change in: susie goals, 
desires, aspirations as 
Rea "2 a result of college 
Academic Skills ( - Grades earned by.:students: 
oe co 8 - "Persistence in colleges 
= Self-report of ability in 


‘math, writing, reading 
and seomprehension | 


Creativity Skills | , - ¢ s in test score that 
; ; ‘measure Originality. and 
’ . ; creative ability 


- Self-report of development 
and activity 


Intellectual Skills , ~ -Change in students’ ‘ability 
to analyze or solve POPs 
lems 


- Measure critical thinking 
activity. by developing . 
an "index score" 


Occupational Skills ‘ - Demonstrated ability to 
. - perform specific tasks 


- Self-report of occupational 


Desires, Aine, Goals 


U 


“yy 
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J iM : - Table 1 (continued) 
“HUMAN CHARACTERISTIC CATEGORIES (continued) 


2750 ; Obtaining a Job br ‘Admission to a \- Percent who received jobs’ 
? fe ~ Follow-up Program - Percent who were accepted 
fo ; : to graduate programs 
2840 Career -and Vocational Roles. - Self-report 
J. . 3170 . Knowledge and Understanding of "= CLEP exam score ¥ 
j Dh eB eteral: Facts oa a - Graduate Record Exam score 
by yo - ; _- Self-report ong 
3120 *-? Knowledge aqedersanting of | 
‘ Be” * 9h we General Processes’ > Comprehension of general ‘« 
a : processes and methods 
ce - Student grades in a general - 
> .Survey course eo 
: 3210 Knowledge -and Understanding of Spec- - Scores on tests that measure 
i ; ialized Facts and Terminology knowledge in specific 
we : fields 
‘ sri 3220 Knowledge- and Understanding of Spec- - Scores on tests that mea- 
\. : ialized Processes : sure knowledge in anata 
as ' fields 
/ 3310 ’ Research and. Scholarship Knowledge ‘) = Number ‘of publications Py 
af? . ' ane Understanding ~ , students 
wg nage + Number of publ ications iy : 
3 ; y, wif faculty | 
3400 Art’ Forms and Works =~ -: Specific accomp]ishments 


in the arts by discipline 
or department . - 


- [ RESOURCE AND SERVICE PROVISION OUTCOME (Code 4000) CATEGORIES 


~ 


™ BI Provision of Facilities ,* - Number of books “in library 


e : ; ; - Number of courts in the 
iY ‘ : at : Recreation Center 


- Number and type of computers 
- 0 > Number of keypunch machines 


4120 Provision or Sponsorship of Events ° - Number of events and type . 
. sponsored by departments 
or university 


: ~ 4210 - is Teaching, ; 4 Handbook on evaluations of 
oe ae oe Specific instructors, — 


4220. Advisory and Analytic Assistance -. Number of advisors avail- 
ae ; on a able to students - 


ey ° . 4 


: ‘ . - Technical assistance avail- 
ie cn . . a "| able to students 


1. 4230 Treatment, Care, and Referral Services - Student health center 
gf ty Met Pe ie EN - Mental health facility 
: | 16-4 9 ~ Personal counsel ing | 
eS Ghee 7 - Day care’ center 


~ 


‘PROCESS NO. I--DEVELOPING OUTCOMES LISTS ee: 


¢ 


Process 1] was used to develop a list of important Outcomes irrespective 
of the current list. Such a list will vary dependina upon the values of the 
planning group making the decision: for example, faculty, students, adminis- Pip 
trators, legislators. However, an overad | institution-wide approach was taken. 

The "audience" focus was limited to, "current students in order. to. limit 
the scope of Process 1. The five major tategorfes of the type-of outcome } 
dimension were oubted: on a sheet of paper that was titled "audience--currently 
enrolled students". Then, under each major category, all subcategories at the 
second and third. jlevels of detail were considered for importance. _ The detailed 
subcategories were then used to stimulate thinking about specific outcomes 
‘that could be considered "essential" or "important". Aiding in this ida 
for. — Subcategory were: (1) A prdduct/eventfoond¥ tian typology, (2) a wathe 
hei erevanee! typology, (3) an: output/imp&ct typology, and (4) and an intended- 
unintended/vatudaSnot valued typology. 

‘ante 2 presents Structure eateguries left out of the original University 
of Colorado listing that were identified during this project as: important | —_ 
categories for the University and its students . Also included are the spdettic: 


priority outcomes thét were identified for each category. 


Tie é he 
IMPORTANT STUDENT OUTCOMES ITEMS IDENTIFIED ‘Fe 
2 ~ THAT WERE NOT ON THE ORIGINAL 
. UNIVERSITY OF COLORADO LIST - 


a 
fos NCHEMS NCHEMS Items to be Added 
*. Category 7 Type-of-Outcome to 
a er Code - a one Category : University of Colorado 
‘ Number . e Name Outcomes List 
i 2 8 s{ ECONOMIC OUTCOME’ (Code 1000) CATEGORIES 
‘1120s, Economic Flexibility and Independence .  - Social Mobility . 
> ‘Tigo Standard of Living _ - ' = Family, assests’ - 
. amy | (other than income) 
HUMAN CHARACTERISTIC OUTCOME (Code 2000) CATEGORIES 
2240 | Expression and Communication Skills | - Self-perception of 
skills Tf the for- 
eign languages 
2260 Interpersonal, Leadership and % ; 
Organization Skills . = sp patoac tive of ‘ 
. interpersonal and 
: : leadership skills” 
. et : : 7 - Positions held in 
ee” 4 pC OW me F organizations that: 
® se pt ‘ require leadership 
2630 Dependability and Responsibility = Self-perception of 
* Sy ities ; .¢ re .dependability and 
. ae . —_ responsibility 
a , : - Employer's op nion 
' an 250 es Flexibility and Adaptability - Self-perception of | 
at 0h So, ae : adaptability : 
Pv ig . oe ‘(other than in job 
5 ee ' , Changes) ae ae 
y 4 KNOWLEDGE, TECHNOLOGY, AND ART FORM 
ps, | OUTCOME (Code 3000) CATEGORIES 
Sue 1330. . . + Knowledge,and. Understanding of General _ = Score on tests measur- - 
. SS - Theory ‘ Ean ing comprehension 
; of general theories 
; 3230 ° Knowledge and Understanding of - Scores on tests measur- , 
Specialized Theory ing theoretical 
: knowledge in specific 
' ( cP » - . fietds 
* -18- 
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CONCLUSION © . #° 4 ) i eT 


As a result of this , tryout of the NCHEMS Outcomes Structure, some impor- . 


"ah tant outcomes avenues were discovered to have been omitted in the original 


list. The Structure served as a reminder of ane breadth of outcome items assoc- 
fated with each outcome eategery: For example, the category MORDSSENES and pe 
Skills" indicated 3 that Expression and Communication Skills such: me aki is in é fh : 
the foreign. ‘languages were important indicators of entering students’ Ability. & 
Basic speaking, writing, and: reading skills’ in the English, language- were con- 
sidered to. be the ‘primary indicators in the orginal outcomes Tisti» As aresult," ‘.. 
‘e 
The final distribution code numbers’ indicated that the Boulder Campus out- 


the list ‘and the Freshman duesttonnaive were modi fied. 


comes list is predominately comprised of fifo characteristics. This is be- 


cause the. variables. in the Model are primarily "student- oriented" outcomes. 
if 


In addition, the classification of unique "audiences" will be atiabte to ‘ 


the Boulder Campus as the outcomes information is incorporated into reports. 


A- public relations staff can categorize outcomes items that would be useful to 
particular audiences. For example, research projects completed by graduate 


students may be of interest to the following’ audiences: 1) current students, 


2) ‘public school districts, 3) industry and business, 4) citizens and policy- 


makers of Colorado. Short executive surimaries written for particular audiences 
= ae % 
‘can be developed which are this informatéon. “ 
The organization which results from the classification into “type Of 
a) 


"outcome" can help campus Teaders and representatives ‘report the outeonas of ae 
educational programs more effectively. Information® can be stored in sompulters 
*4zed student. information systems keyed to the NCHEMS classification’ codes. 

The classification system can also be used to.organize reference materials 


collécted on outcome variables. 
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